| Author |
Message |
|
|
Post subject: missing feature series #1: control over file transfer
 Posted: Dec 01, 2008 - 07:45 PM
|
|
Joined: Oct 28, 2008
Posts: 66
Status: Offline
|
|
Following slam advice, I'm making a series of distinct topics to search for certains features I couldn't find under linux (I'm using debian testing / sidux with kde). This is the first one of the series and it's about file transfer.
The point being to give the user control over his file transfers so he can arrange them anyway he likes in order to optimize ressources and user experience.
If you are familiar with ftp clients, what I'm gonna talk about are basic features that have been around for a while now:
update: it seems a lot of people are confused by the mention of ftp clients and miss the point, so here is a little clarification: I'm talking about missing features that should be provided by the underlying system for any kind of file transfers. Think about file managers providing or more exactly *not* providing the same kind of features a ftp client has only for each and every file transfer. Think drag'n'drop in konqueror/nautilus/krusader/thunar/dolphin/... on a box with no network connectivity at all.
/update.
1- giving the choice to resume an incomplete file
2- giving the possibility to pause/resume a transfer
3- offering to queue a transfer instead of instantly starting it
4- giving the possibility to limit the speed of a transfer
5- offering queue management (add/delete/rearrange/modify)
Some more details:
1) I have this large backup file (100GB) I'm copying from B to A, a power shortage later and I end up with a 88GB being copied and having to start the whole process from scratch. Why overwrite those first 88GB when they are perfectly valid and not just complete the missing 12GB ?
2) So I restarted the file transfer from scratch and it has now copied 45GB out of the 100GB when my friend joe just drop in a hurry with his external drive containing vital data (10GB) for our project that I need to copy now!, I plug his drive in (let's call it C) and start copying the data from C to A and it competing with the other transfer for ressources, Joe cannot wait much longer (he has a train to catch). Why stopping the transfer from B to A when pausing it would do the trick ?
3) Transfer from joe's drive's done, Joe's gone and I started my transfer from B to A from scratch *again*. Enters Sarah who need parts of the data I copied from joe's drive copied to her thumbdrive (let's call it D). So we go through the data with Sarah and she happens to need several files and directories that are located in various places of the directory structure and her thumbdrive has not enough space for the whole data. So I select files from each location in A and start transfering them to D one location after another, after a short while the file transfers are crawling to a snail pace because they're all competing for ressources. Why not offer the possibility to add each new transfer to a queue which allows for optimal use of ressources ?
4) So we are forced to wait for a transfer to finish before starting a new one, and I end up showing picture of my trip to a foreign place to Sarah while waiting and she likes them and would like a copy, but she doesn't have enough space on her thumbdrive, so I'm going to burn her a CD. Unfortunately the burning process failed because the drive couldn't output data fast enough due to the others file transfers. Why not offer the possibility to limit speed on other file transfers to prioritize the important one ?
5) If you need an example or more details about the queue management, feel free to ask, but I think it's quite obvious.
This kind of shortcomings seems to affect other OS as well, maybe a old reminder from a distant past, but 3rd party software exist under windows exist to address this shortcoming and provide this kind of functionality. As a reference you can check supercopier, teracopy, totalcopy, total commander.
update:
Ongoing discussion about adding these to ubuntu.
A quick-and-dirty temporary solution here: minicopier, which tries to be a java clone of supercopier.
After running some test of minicopier, it is more of a proof of concept and isn't suitable for daily use (at least for me) due its lack of integration with the system and not-so practical interface, but can make for an interesting last resort emergency solution as it is portable.
This demonstrates that the solution has to come from a lower level, maybe from the kernel itself ?
update #2: According to this arstechnica article, there will be a plasmoid in kde4 consisting of a panel stacking all kio based file operations with two buttons stop and play/pause. (screenshot). |
Last edited by JohnyDoey on Dec 09, 2008 - 01:15 PM; edited 5 times in total
|
| |
|
|
|
 |
|
|
|
Post subject: RE: missing feature series #1: control over file transfer
Posted: Dec 01, 2008 - 08:07 PM
|
|
Joined: Apr 30, 2008
Posts: 25
Location: AT - Tirol
Status: Offline
|
|
To resume a file copy that got interrupted use curl (displays a nice progress report as well).
curl -C - -O file:///media/somefile |
|
|
| |
|
|
|
|
|
|
|
Post subject: RE: missing feature series #1: control over file transfer
Posted: Dec 01, 2008 - 08:08 PM
|
|
Joined: Jul 17, 2007
Posts: 30
Location: Oklahoma
Status: Offline
|
|
| Have you tried the extension for Firefox, FireFTP? |
|
|
| |
|
|
|
|
|
|
|
Post subject: RE: missing feature series #1: control over file transfer
Posted: Dec 01, 2008 - 09:22 PM
|
|
Team Member
Joined: Nov 24, 2006
Posts: 2082
Location: w3
Status: Offline
|
|
I am moving around many Tbs a week between servers and local machines, and for this very kind of problems I love wget, but also rsync - but both might be considered not beginner friendly because they are CLI tools, and rsync needs a server daemon to run. Also I do not have the need of limiting bandwidth used (while I remember that wget is able to do that).
As the data contained in those big files is often not my own, security is very important so I like to have the possibility to use both tools via SSH and/or VPN.
This might be not exactly the usage pattern others need, so please continue to post folks.
Greetings,
Chris |
_________________
an operating system must operate - sidux.com
development is life - code.zikula.org
|
| |
|
|
|
|
|
|
|
Post subject: RE: missing feature series #1: control over file transfer
Posted: Dec 01, 2008 - 10:34 PM
|
|
Joined: Oct 28, 2008
Posts: 66
Status: Offline
|
|
Let's focus more on desktop use and things like drag'n'drop in konqueror.
I've just found an ongoing discussion about adding these to ubuntu.
And maybe a quick-and-dirty temporary solution here: minicopier, which seems to be java version of supercopier.
I just ran a few tests with minicopier, it lacks some of the features (limiting speed, modifying queue, providing info about actual speed and estimated remaining time), made konqueror crash a couple times, doesn't integrate whatsoever with the system and is not practical for daily use (drag things one after another onto a basket icon and then drag the destination onto another icon), but it is a working proof of concept. |
Last edited by JohnyDoey on Dec 01, 2008 - 11:08 PM; edited 4 times in total
|
| |
|
|
|
|
|
|
|
Post subject: RE: missing feature series #1: control over file transfer
Posted: Dec 01, 2008 - 10:35 PM
|
|
Joined: Nov 27, 2006
Posts: 547
Location: Griffith NSW Australia
Status: Offline
|
|
from man rsync:
Code:
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
rsync also provides the resume capabilities, as well as rolling checksum capabiliites so that if you are regularly copying largely unchanged data (eg backups) then only the changed data is transferred over the network |
|
|
| |
|
|
|
|
|
|
|
Post subject: RE: missing feature series #1: control over file transfer
Posted: Dec 01, 2008 - 10:39 PM
|
|
Joined: Nov 30, 2006
Posts: 617
Location: USA
Status: Offline
|
|
|
|
|
|
|
|
|
Post subject: Re: missing feature series #1: control over file transfer
Posted: Dec 01, 2008 - 11:14 PM
|
|
Joined: Nov 30, 2006
Posts: 3610
Location: Budapest
Status: Offline
|
|
|
JohnyDoey wrote:
1- giving the choice to resume an incomplete file
2- giving the possibility to pause/resume a transfer
3- offering to queue a transfer instead of instantly starting it
4- giving the possibility to limit the speed of a transfer
5- offering queue management (add/delete/rearrange/modify)
kftpgrabber
gftp (this is a gtk2 application, so rather for GNOME but runs in KDE as well)
hubi |
_________________
Bonitas stultitiaque sodales sunt.
|
| |
|
|
|
|
|
|
|
Post subject: RE: Re: missing feature series #1: control over file transfe
Posted: Dec 02, 2008 - 02:28 AM
|
|
Joined: Dec 02, 2006
Posts: 1060
Location: East Coast
Status: Offline
|
|
As slam said before, for this kind of things I use wget, or a downloader manager ( dTa firefox extension ). However, this only works from point A to X, not from B to A, or C to A, since A is the only one with apache on.
I can't remember if Filezilla can resume transfers
Can wget -Nc be use in ftp? |
|
|
| |
|
|
|
|
|
|
|
Post subject: RE: Re: missing feature series #1: control over file transfe
Posted: Dec 02, 2008 - 02:42 AM
|
|
Joined: Oct 28, 2008
Posts: 66
Status: Offline
|
|
How come you are all talking about ftp client and network applications when I'm talking about the lack of these usual network features for non-network operations ?
Have I explained myself so badly ? |
|
|
| |
|
|
|
|
|
|
|
Post subject: Re: RE: Re: missing feature series #1: control over file tra
Posted: Dec 02, 2008 - 03:16 AM
|
|
Joined: Nov 27, 2006
Posts: 547
Location: Griffith NSW Australia
Status: Offline
|
|
|
JohnyDoey wrote:
How come you are all talking about ftp client and network applications when I'm talking about the lack of these usual network features for non-network operations ?
Have I explained myself so badly ?
rsync can be used over a network, or not. |
|
|
| |
|
|
|
|
|
|
|
Post subject: RE: Re: RE: Re: missing feature series #1: control over file
Posted: Dec 02, 2008 - 04:11 AM
|
|
Joined: Mar 04, 2008
Posts: 24
Status: Offline
|
|
I'm reluctant to feed the troll, but in the hopes others will read this and actually think about the responses here we go:
Your primary complaint here seems to be with the drag and drop functionality of certain file-managers. That's fine. But that said, what do you hope to accomplish by posting at a forum used by a fraction of a percentage of Linux users world wide, a forum intended as technical support for a small community distro based of another larger community distro, and a forum that is not likely to be visited by anyone working in development for these major projects about these issues?
This is a wish-list bug. Nothing more. You are welcome to contribute the KDE or Gnome projects if you feel this level of functionality should be available within their file-managers. You are welcome to visit those developer's forums and discuss ideas there. You are welcome to file wish-list bugs at the project level. You are welcome to take the tools and knowledge that the linux community gives you free of charge and build your own tool. But this discussion is pointless, particularly since you don't seem to want to hear the respondents suggestions, rebuttals, and advice. |
|
|
| |
|
|
|
|
|
|
|
Post subject: Re: RE: Re: RE: Re: missing feature series #1: control over
Posted: Dec 02, 2008 - 08:48 AM
|
|
Team Member
Joined: Nov 24, 2006
Posts: 2082
Location: w3
Status: Offline
|
|
|
devious wrote:
...But that said, what do you hope to accomplish by posting at a forum used by a fraction of a percentage of Linux users world wide, a forum intended as technical support for a small community distro based of another larger community distro, and a forum that is not likely to be visited by anyone working in development for these major projects about these issues?...
While I fully agree with everything else you said, my personal experience is that a lot of upstream developers are using sidux, and visit this forum. Talking about this here might also initiate interest for those upstream projects with existing sidux users, and they might start contributing to those projects. So, sidux might be not the big mother of all Linux distros  - but it's user base is usually better skilled and more interested/active in development than in other distros.
I also remember that similar discussions in here have led to one or another sidux developer jumping in and creating something - if the idea was presented clearly. E.g. look at Ceni, the sidux installer, the sidux control center, Hermes, and many other applications with their birth place here.
Said that, did anyone already look into the recent developments KDE4 has pushed in related projects?
Greetings,
Chris |
_________________
an operating system must operate - sidux.com
development is life - code.zikula.org
|
| |
|
|
|
|
|
|
|
Post subject: Re: RE: Re: missing feature series #1: control over file tra
Posted: Dec 02, 2008 - 09:14 AM
|
|
Joined: Nov 30, 2006
Posts: 3610
Location: Budapest
Status: Offline
|
|
|
JohnyDoey wrote:
How come you are all talking about ftp client and network applications when I'm talking about the lack of these usual network features for non-network operations ?
Have I explained myself so badly ?
You want speed transfer limits for copying from one device to another? From one partition to another? From one folder to another?
I am getting bored here.
hubi |
_________________
Bonitas stultitiaque sodales sunt.
|
| |
|
|
|
|
|
|
|
Post subject: RE: Re: RE: Re: missing feature series #1: control over file
Posted: Dec 02, 2008 - 09:30 AM
|
|
Joined: Dec 21, 2007
Posts: 182
Status: Offline
|
|
Also it might be very rare that such a scenario, what you describe above, might happen.
The already possible solutions might be enough for 95% of the users.
The reason that such a feature isn't implemented may be due to the reason that there is no good relation between effort and additional benefit.
Most of the people would'nt need something like this. |
|
|
| |
|
|
|
|
|
|
