The first computers and simple calculators were only 4 bit. This essentially means they could only index 2^4 = 32 unique entries in a table using only those 4 bits. It specifically describes the so called 'width' of the address bus. A bus is simply a set of parallel wires that convey signals from one place in the computer to another. Clearly, a wider bus will permit you to move more data at once, and also address more objects with a single index. Specifically, the address bus is important because this dictates how much memory you can address using one data-transfer of an address. This address is really an index into memory. So an 8-bit computer could address a memory size of 2^8=256 bytes. This is still very restrictive, so engineers quickly doubled that to 32 bits which addresses a more respectable 2^32=4 294 967 296 bytes. The complexity of the CPU and motherboards tends to rise as the bus width is increased and the 32 bus has persisted for many years.
A 64 bit computer doubles this again permitting memory sizes that are addressable in one address to 2^64 = 1.8 x 10^19 which is a very large space.
Programs compiled to run on 64 bit computers will not run on 32 bit computers, but a program compiled to run on a 64 bit computer can often run by using a compatibility mode or translator.
In broad terms, providing all the drivers and software works for you, a 64 bit computer will allow much more memory to be available to an operating system and its programs, and should run faster and more efficiently than a 32 bit system.