GDDR3 and GDDR5 actually have the same raw speed for a given bus width and clock. The difference is that GDDR3 can only transfer in one direction per clock cycle, while GDDR5 can transfer in both directions. So if you read a bunch of data from VRAM or write a bunch of data to VRAM, GDDR3 and GDDR5 will look the same, all else being equal. That's not what actually happens when rendering, though. There's always a simultaneous need to read and write memory. GDDR3 has to take turns, while GDDR5 can perform them simultaneousl. So the overall memory throughput for GDDR5 is twice that of GDDR3, given the same bus width and clock speed.
It's rather like half-duplex and full-duplex with a network switch. If you're connected at 1Gbps at half-duplex, you can send data at 1Gbps or receive data at 1Gbps, but not at the same time. So if you're trying to send and receive at the same time, the aggregate speed of a 1Gbps full-duplex link is 2Gbps - still 1Gbps in either direction, though. In contrast, if sending and receiving simultaneously with a half-duplex connection, the one-way transfer speed would be 0.5Gbps.
In your specific case, you have a 9600 GT with a 256-bit memory bus at 900MHz, which means 57.6Gbps total theoretical throughput (i.e. read and write combined). Then you moved to a 7770 which has a 128-bit memory bus at 1125MHz, which means 72.0Gbps total throughput. In any kind of test which involved only reading or only writing, the GDDR3 would be faster - 57.6Gbps max versus 36Gbps max with GDDR5. Under real world conditions, the GDDR5 will be faster.
In rendering power, the 7770 should be considerably faster as well. It's hard to find any common benchmarks between the two given the age difference, but it looks like a factor of two to three difference.