[Hackers]

Have you ever had to push a set of 1000+ files? You know how tedious it can be to have a commit or push fail with a large amount of files.
This is a bash script that will add, commit, and push 200 files at a time. You can obviously increase or decrease this amount if you want.

note: if a single file fails, the script will keep going which may be a problem if you lose a batch commit/push - keep an eye on it, and understand that this is just a script I used that worked for me and you must use it at your own risk. For example if you don't have LFS enabled for large files, or another issue pops up then this script won't solve for that.

Code

#!/bin/bash

# Define the branch name
branch_name="my-new-branch-name-goes-here"

# Get the list of unstaged files
files_to_push=$(git ls-files --others --modified --exclude-standard)

# Check if there are files to push
if [ -z "$files_to_push" ]; then
  echo "No files to push."
  exit 0
fi

# Convert the list of files into an array
files_array=($files_to_push)

# Define the number of files to push at a time
batch_size=200

# Get the total number of files
total_files=${#files_array[@]}

# Calculate the number of batches needed
num_batches=$(( (total_files + batch_size - 1) / batch_size ))

# Push files in batches
for (( i=0; i<num_batches; i++ ))
do
  start_index=$(( i * batch_size ))
  end_index=$(( start_index + batch_size - 1 ))

  if [ $end_index -ge $total_files ]; then
    end_index=$(( total_files - 1 ))
  fi

  # Get the files for the current batch
  batch_files=${files_array[@]:$start_index:$(( end_index - start_index + 1 ))}

  # Add only the files in the current batch
  git add ${batch_files[@]}

  # Commit the batch
  git commit -m "Batch commit $((i+1))"

  # Push the batch to the specified branch
  git push origin $branch_name
done

echo "All files pushed successfully in batches of $batch_size."

[secret_code]

Have you ever had to push a set of 1000+ files? You know how tedious it can be to have a commit or push fail with a large amount of files.
This is a bash script that will add, commit, and push 200 files at a time. You can obviously increase or decrease this amount if you want.

note: if a single file fails, the script will keep going which may be a problem if you lose a batch commit/push - keep an eye on it, and understand that this is just a script I used that worked for me and you must use it at your own risk. For example if you don't have LFS enabled for large files, or another issue pops up then this script won't solve for that.

Code

#!/bin/bash

# Define the branch name
branch_name="my-new-branch-name-goes-here"

# Get the list of unstaged files
files_to_push=$(git ls-files --others --modified --exclude-standard)

# Check if there are files to push
if [ -z "$files_to_push" ]; then
  echo "No files to push."
  exit 0
fi

# Convert the list of files into an array
files_array=($files_to_push)

# Define the number of files to push at a time
batch_size=200

# Get the total number of files
total_files=${#files_array[@]}

# Calculate the number of batches needed
num_batches=$(( (total_files + batch_size - 1) / batch_size ))

# Push files in batches
for (( i=0; i<num_batches; i++ ))
do
  start_index=$(( i * batch_size ))
  end_index=$(( start_index + batch_size - 1 ))

  if [ $end_index -ge $total_files ]; then
    end_index=$(( total_files - 1 ))
  fi

  # Get the files for the current batch
  batch_files=${files_array[@]:$start_index:$(( end_index - start_index + 1 ))}

  # Add only the files in the current batch
  git add ${batch_files[@]}

  # Commit the batch
  git commit -m "Batch commit $((i+1))"

  # Push the batch to the specified branch
  git push origin $branch_name
done

echo "All files pushed successfully in batches of $batch_size."

Nice one, but you might want to update the title using "bash" instead of "batch", batch is used for windows scripts while bash for linux shell scripts, does it let u supply additional flags?

[Hackers]

Nice one, but you might want to update the title using "bash" instead of "batch", batch is used for windows scripts while bash for linux shell scripts, does it let u supply additional flags?

It's a bash script that allows you to add/commit/push files in batches.
I understand the confusion though, but the title is NOT referring to batch / .bat

[secret_code]

It's a bash script that allows you to add/commit/push files in batches.
I understand the confusion though, but the title is NOT referring to batch / .bat

now i understand, i was confusing, right, this is a useful script for automating boring github stuff... thanks

[Hackers]

now i understand, i was confusing, right, this is a useful script for automating boring github stuff... thanks

yeah pretty much lol.
I did update the thread title - good call, didn't think of that being misinterpreted.

I did make 1 tweak, now it skips over a push if there is an error, moving on to the next push in the batches. This way if any single part can't be pushed, you can figure it out manually after instead of losing those changes (like in v1)

Code

#!/bin/bash

# Define the branch name
branch_name="personalBranchName"

# Get the list of unstaged files
files_to_push=$(git ls-files --others --modified --exclude-standard)

# Check if there are files to push
if [ -z "$files_to_push" ]; then
  echo "No files to push."
  exit 0
fi

# Convert the list of files into an array
files_array=($files_to_push)

# Define the number of files to push at a time
batch_size=200

# Get the total number of files
total_files=${#files_array[@]}

# Calculate the number of batches needed
num_batches=$(( (total_files + batch_size - 1) / batch_size ))

# Function to push a batch
push_batch() {
  local start_index=$1
  local end_index=$2

  if [ $end_index -ge $total_files ]; then
    end_index=$(( total_files - 1 ))
  fi

  # Get the files for the current batch
  batch_files=${files_array[@]:$start_index:$(( end_index - start_index + 1 ))}

  # Add only the files in the current batch
  git add ${batch_files[@]}

  # Commit the batch
  git commit -m "Batch commit $((i+1))"

  # Push the batch to the specified branch
  if ! git push origin $branch_name; then
    echo "Push failed for batch $((i+1)), undoing commit..."
    git reset HEAD~1
    echo "Commit undone. Moving on to the next batch."
  fi
}

# Push files in batches
for (( i=0; i<num_batches; i++ ))
do
  start_index=$(( i * batch_size ))
  end_index=$(( start_index + batch_size - 1 ))
  
  push_batch $start_index $end_index
done

echo "All files pushed successfully in batches of $batch_size."

Also, here is a similar script that I made a few weeks ago that is similar, but it only does the add & commit without the push, because I hit a snag in the commit process but pushing was fine. Here's that script if you only want to add & commit. This uses a uniquely different method, pushes all filepaths to an all_files.txt file first, then it loops through 500 files at a time and makes batch_# files, and then it processes from there. I prefer the current script in this thread here, but in case it's helpful to someone - here:
Spoiler (Click to View)

Code

#!/bin/bash

# Generate the list of files to be staged
git status --porcelain | grep '^??' | cut -c 4- > all_files.txt

# Split the file list into batches of 500
split -l 500 all_files.txt batch_

# Loop through the batches to stage and commit
for batch in batch_*; do
    git add $(cat $batch)
    git commit -m "Staging batch $(basename $batch)"
done

# git push origin personalBranchName

# Clean up batch files
rm batch_*
rm all_files.txt

[secret_code]

yeah pretty much lol.
I did update the thread title - good call, didn't think of that being misinterpreted.

I did make 1 tweak, now it skips over a push if there is an error, moving on to the next push in the batches. This way if any single part can't be pushed, you can figure it out manually after instead of losing those changes (like in v1)

Code

#!/bin/bash

# Define the branch name
branch_name="backup-live-5-31-2024"

# Get the list of unstaged files
files_to_push=$(git ls-files --others --modified --exclude-standard)

# Check if there are files to push
if [ -z "$files_to_push" ]; then
  echo "No files to push."
  exit 0
fi

# Convert the list of files into an array
files_array=($files_to_push)

# Define the number of files to push at a time
batch_size=10

# Get the total number of files
total_files=${#files_array[@]}

# Calculate the number of batches needed
num_batches=$(( (total_files + batch_size - 1) / batch_size ))

# Function to push a batch
push_batch() {
  local start_index=$1
  local end_index=$2

  if [ $end_index -ge $total_files ]; then
    end_index=$(( total_files - 1 ))
  fi

  # Get the files for the current batch
  batch_files=${files_array[@]:$start_index:$(( end_index - start_index + 1 ))}

  # Add only the files in the current batch
  git add ${batch_files[@]}

  # Commit the batch
  git commit -m "Batch commit $((i+1))"

  # Push the batch to the specified branch
  if ! git push origin $branch_name; then
    echo "Push failed for batch $((i+1)), undoing commit..."
    git reset HEAD~1
    echo "Commit undone. Moving on to the next batch."
  fi
}

# Push files in batches
for (( i=0; i<num_batches; i++ ))
do
  start_index=$(( i * batch_size ))
  end_index=$(( start_index + batch_size - 1 ))
  
  push_batch $start_index $end_index
done

echo "All files pushed successfully in batches of $batch_size."

Also, here is a similar script that I made a few weeks ago that is similar, but it only does the add & commit without the push, because I hit a snag in the commit process but pushing was fine. Here's that script if you only want to add & commit. This uses a uniquely different method, pushes all filepaths to an all_files.txt file first, then it loops through 500 files at a time and makes batch_# files, and then it processes from there. I prefer the current script in this thread here, but in case it's helpful to someone - here:
Spoiler (Click to View)

Code

#!/bin/bash

# Generate the list of files to be staged
git status --porcelain | grep '^??' | cut -c 4- > all_files.txt

# Split the file list into batches of 500
split -l 500 all_files.txt batch_

# Loop through the batches to stage and commit
for batch in batch_*; do
    git add $(cat $batch)
    git commit -m "Staging batch $(basename $batch)"
done

# git push origin backup-live-5-16-2024

# Clean up batch files
rm batch_*
rm all_files.txt

nice job dude, the one from this thread is better then the old one

[Hackers]

nice job dude, the one from this thread is better then the old one

oh yeah definitely.
the old one got the job done at the time but I was not a fan of how it created the batch_# files and also with it not pushing until the end it didn't make much sense (I got lucky and my pushes went through last time I needed this lol)

I forget why I didn't push in the loop but I moved it out & commented it out for a reason couple weeks ago

[Pinhead Larry]

Ever heard of Git Desktop?

Jk cool stuff though.

[Hackers]

Ever heard of Git Desktop?

Jk cool stuff though.

I'll never go back to GUI Git desktop - it's so much easier and faster to manage in CLI in my opinion. It also is nice to have a history of commands, etc.
I also have noticed a trend where anyone who can use git CLI is more than capable of using Git Desktop, but the people that only use Git Desktop have no idea how to use the CLI for it. I'd rather be in category1 to be honest.

[Dirty]

I'll never go back to GUI Git desktop - it's so much easier and faster to manage in CLI in my opinion. It also is nice to have a history of commands, etc.
I also have noticed a trend where anyone who can use git CLI is more than capable of using Git Desktop, but the people that only use Git Desktop have no idea how to use the CLI for it. I'd rather be in category1 to be honest.

In general with obviously some diminishing returns, the more you use CLIs and the less and less you rely on your keyboard the faster you are. Even if you add in the initial time investment and slowdown to learn.

[Hackers]

In general with obviously some diminishing returns, the more you use CLIs and the less and less you rely on your keyboard the faster you are. Even if you add in the initial time investment and slowdown to learn.

yeah, exactly and you can keybind commands too.
It's also just good to know what exactly is going on behind GUIs. It's the same logic (for this here) as I used when I used GCC CLI to compile C++ instead of using Visual Studio. It was a pain in the ass ,but I know a whole lot more about compiling options than I ever would in a GUI.

[Dirty]

yeah, exactly and you can keybind commands too.
It's also just good to know what exactly is going on behind GUIs. It's the same logic (for this here) as I used when I used GCC CLI to compile C++ instead of using Visual Studio. It was a pain in the ass ,but I know a whole lot more about compiling options than I ever would in a GUI.

I also can't imagine being productive when working with containers, kubernetes,git and all sorts of other tools and doing it all through different GUIs instead of opening 2 or more terminals and doing everything cleanly in a singular window

[Hackers]

I also can't imagine being productive when working with containers, kubernetes,git and all sorts of other tools and doing it all through different GUIs instead of opening 2 or more terminals and doing everything cleanly in a singular window

yeah exactly... also I set up different origins a lot of times so I can toggle between the main origin repo & a fork repo / similar.
It's such an efficiency to be able to jump around repos like that within 1 project

I agree with you completely. I once was a GUI warrior years ago but after years of Linux (and windows, powershell, etc) CLI, I just can't see myself going back.