Surprisingly i don't see any mention of the good old Midnight Commander here. To my mind, it's probably the most universal & usefull file manager for shell with power-capabilities, one of the "must have" tools for the case, which is also allowing you to connect through SSH, FTP, SFTP as well, while on the second panel you can open any other (your local) filesystem and so work with files freely.
All you need is: apt-get install mc (from universe)
After that, run mc, open the menu for left or right panel, choose shell connection, enter username@remote-ip, the password - actually, that's it.. copy (here: download) files/folders from one machine to another with F5, move with F6, and so on according to buttons below. For the old MS-users: just like in NC for DOS