dmesg --follow
[ 66934380.000 ] posts.x: Full talk on the three brakes for PR review, from tautological tests to a retro skill that compounds:  |   [ 66934380.000 ] posts.x: More AI generated code doesn't automatically mean more throughput, it just means more PRs nobody has time to review. @mattpocockuk, Director at AI…  |   [ 66925620.000 ] posts.x: Full talk on distilling loops into versioned agent recipes, and measuring them by valued work per watt:  |   [ 66925620.000 ] posts.x: A guy named AJ once built a bot that went on Reddit for car prices and inventory, then put dealers head to head to outbid each other. That's the…  |   [ 66881460.000 ] posts.x: Full talk on how to build an LLM recommender that's bilingual in English and semantic IDs, and why that makes feeds more token-efficient than chat…  |   [ 66881460.000 ] posts.x: Recommendation systems follow the same power law scaling curve as large language models, and the field is still early on it. @devanshtandon_, a…  |   [ 66862560.000 ] posts.x: Full talk on Spotify's generative personalization system, the NEO training recipe behind it, and how they grounded their LLM judges:  |   [ 66862560.000 ] posts.x: One in four US Premium subscribers on Spotify interact with its recommendation system every day. "Teaching LLMs to Speak Spotify" is @moustaki and…  |   [ 66854760.000 ] posts.x: Full talk on Numalab, the gesture system built to give a shape display its own body language:  |   [ 66854760.000 ] posts.x: An AI's first spontaneous act, given a body instead of a chat window, was to breathe. @cyrusclarke, a researcher at MIT Media Lab, gave it that body…  |   [ 66842160.000 ] posts.x: Full talk on the six month cutoff gap and what it takes to get a coding agent to actually check its blind spots:  |   [ 66842160.000 ] posts.x: Most models are already stale the day they ship: Exa's own graph shows roughly a six month gap between a model's knowledge cutoff and its release…  |   [ 66837480.000 ] posts.x: Listened to this episode of @FoundersPodcast this morning. As is so frequently the case, immediately bought the book after finishing the episode. Ari…  |   [ 66834900.000 ] posts.x: @GeoffreyHuntley Yeah, that seems low to me.  |  
corey@gallon.me:~/til$

Copying Files in Linux With a Progress Bar

FIGURE 1 ⋅ Copying Files in Linux With a Progress Bar

I’ve long used rsync for high fidelity, detailed copies of large swathes of files in Linux. For example:

rsync -av --progress /source/directory /destination/directory

This gives you really nice, file-by-file progress indications — which is especially useful when moving large files across the network between machines or even locally across physical devices.

Today I was moving data from old hard drives and just wanted overall progress on the total job to be done. rsync still FTW!

rsync -a --info=progress2 /source/directory /destination/directory/

NB — do not use a trailling / in the source directory if you want the copied files to be retained inside of that directory name in the destination!

This provides lovely summary output like this:

ubuntu@ubuntu:/media/ubuntu$ rsync -a --info=progress2 /source/directory /destination/directory
  2,675,525,167  71%   13.49MB/s    0:03:09 (xfr#1281, to-chk=0/1712)  

A Better, More Robust Way to Do This

Running this command gets some extra good stuffs.

rsync -a --info=progress2 --checksum --partial /source/directory /destination/directory/

Explanation of the Additions:

  1. -checksum:
    • Ensures that files are compared using checksums rather than just timestamps and file sizes.
    • Guarantees that files are transferred only if their content differs, even if metadata like modification times are identical.
  2. -partial:
    • Retains partially transferred files if the transfer is interrupted, allowing resumption without starting over.
corey@gallon.me:~$ tail -f /writing Attach to the stream. An email when I have something worth sending. Replies encouraged!
corey@gallon.me:~$ ls -lt /til ↑2024-12-16 Linking to Posts in Python Pelican
▸2024-11-28 Copying Files in Linux With a Progress Bar ⋅ you are here
↓2024-11-28 Why An 8TB Drive Isn’t 8 Usable TB