Author Archives: admin

Delete old files from huge directory with perl

If you need to delete old files from a directory but ls/find/rm/etc hang because of the number of files (especially if they’re on a slow NFS share), then you can use perl!

chdir "DIRECTORY" or die;
opendir D, ".";
$count = 0;
while ($n = readdir D) {
    $days = -M $n;
    if ($days > 3){
        #print "deleting $n $days\n";
        unlink $n or warn "Can't unlink $file: $!";
print "deleted $count files\n";

Replace DIRECTORY with your directory name, and 3 with the max age of the files to keep (here it will keep files 3 days old or less).

If you’re running this on AWS EFS then temporarily switch the filesystem throughput mode from burstable to provisioned, otherwise it will start quick but get very very slow once you’re out of burstable credits.

Access custom phinx variable inside seed/migration

Any variable passed in the environment config can be accessed from a seed/migration (after run()) through the adapter options $this->getAdapter()->getOptions().

For example, the phinx.php:

return [
    'environments' => [
        'default_migration_table' => 'phinxlog',
        'default_database' => 'production',
                'production' => [
                    'adapter' => 'mysql',
                        'name' => 'proddb',
                        'host' => 'localhost',
                        'user' => 'proddb',
                        'pass' => 'password',
                        'custom_thing' => $custom_thing,

Then in the migration/seed:


Buildbot custom BuildStep with renderables example

Custom buildsteps need to:
* Accept and take any arguments in the __init__ method
* Call the parent class’ __init__ method
* Declare any attributes that need to be rendered in renderables attribute (this happens after __init__ and before run
* Do the action in the run method and return SUCCESS from this method

Here’s a simple example CustomStep designed to pass arguments directly to another class AnotherClass and call a method on the instance.

from buildbot.process.buildstep import BuildStep
from buildbot.process.results import SUCCESS

class CustomStep(BuildStep):
    name = 'customstep'
    description = 'Do the Thing'
    descriptionDone = 'Done the Thing'

    kwargs = None
    renderables = ['kwargs']

    rendered = True  # attributes are rendered

    def __init__(self, **kwargs):
        self.kwargs = kwargs
        super(CustomStep, self).__init__()

    def run(self):
        c = AnotherClass(**self.kwargs)
        return SUCCESS

And to run the step:

    'id': util.Property('id'),
    'image': util.Property('image'),
    'source': 'someplace',
    'url': util.Interpolate("https://%(prop:branch)"),

If your build raises an exception on the new buildstep but gives no output, check the buildbot master logs. If you get an error TypeError: list indices must be integers, not NoneType with this line: stepsumm += u' (%s)' % Results[self.results] then make sure your run returns SUCCESS.

Mysql service shows stopped when mysql is running after upgrade

In this case, the upgrade changed a line in the service init script /etc/init.d/mysql from if ! mysqladmin ping &>/dev/null;then to if ! $(mysqladmin --no-defaults --socket="$socket" --user=UNKNOWN_MYSQL_USER ping &>/dev/null) ;then.

Printing out $socket revealed it was empty. That value was coming from /usr/sbin/mysqld --print-defaults, which prints out the the configuration defined in the config file. I had not defined socket in my config file (instead just using mysql default socket location), so adding the line socket=/var/run/mysqld/mysqld.sock to /etc/mysql/my.cnf solved the problem.

Eloquent HasManyThrough 5.5 functionality with version 5.1

The 5.1 version of HasManyThrough is limited; you cannot specify the key on the final table that connects it to the intermediate table.

If you’re unfortunately stuck on 5.1 but want this functionality, the following HasManyThrough should help. Tested with normally loading the relationship (ie modelInstance->relationship) and using with (ie model::with('relationship')->where()), not yet with has or whereHas.

use Illuminate\Database\Eloquent\Model as Eloquent;
use Illuminate\Database\Eloquent\Relations\Relation;
use Illuminate\Database\Eloquent\Relations\HasManyThrough;

 * A custom version of HasManyThrough upgrading Eloquent 5.1 HasManyThrough to the 5.5 version.
 * This allows us to specify the key on the intermediate table connecting to the final table.
class CustomHasManyThrough extends HasManyThrough {

     * The "through" parent model instance.
     * @var \Illuminate\Database\Eloquent\Model
    protected $throughParent;

     * The local key on the intermediary model.
     * @var string
    protected $secondLocalKey;

     * Create a new has many through relationship instance.
     * @param  \Illuminate\Database\Eloquent\Builder  $query
     * @param  \Illuminate\Database\Eloquent\Model  $farParent
     * @param  \Illuminate\Database\Eloquent\Model  $throughParent
     * @param  string  $firstKey
     * @param  string  $secondKey
     * @param  string  $localKey
     * @param  string  $secondLocalKey
     * @return void
    public function __construct($query, $farParent, $throughParent, $firstKey, $secondKey, $localKey, $secondLocalKey){
        $this->localKey = $localKey;
        $this->firstKey = $firstKey;
        $this->secondKey = $secondKey;
        $this->secondLocalKey = $secondLocalKey;

        $this->farParent = $farParent;        
        $this->throughParent = $throughParent;

        Relation::__construct($query, $throughParent);

     * Get the fully qualified parent key name.
     * @return string
    public function getQualifiedParentKeyName(){
        return $this->parent->getTable().'.'.$this->secondLocalKey;

     * Set the constraints for an eager load of the relation.
     * @param  array  $models
     * @return void
    public function addEagerConstraints(array $models)
        $table = $this->parent->getTable();
        $this->query->whereIn($table.'.'.$this->firstKey, $this->getKeys($models, $this->localKey));

     * Match the eagerly loaded results to their parents.
     * @param  array   $models
     * @param  \Illuminate\Database\Eloquent\Collection  $results
     * @param  string  $relation
     * @return array
    public function match(array $models, \Illuminate\Database\Eloquent\Collection $results, $relation)
        $dictionary = $this->buildDictionary($results);
        // Once we have the dictionary we can simply spin through the parent models to
        // link them up with their children using the keyed dictionary to make the
        // matching very convenient and easy work. Then we'll just return them.
        foreach ($models as $model) {
            $key = $model->getAttribute($this->localKey);
            if (isset($dictionary[$key])) {
                $value = $this->related->newCollection($dictionary[$key]);
                $model->setRelation($relation, $value);
        return $models;

class CustomEloquent extends Eloquent {

     * Replace the Eloquent has-many-through relationship with our custom one.
     * @param  string  $related
     * @param  string  $through
     * @param  string|null  $firstKey
     * @param  string|null  $secondKey
     * @param  string|null  $localKey
     * @return \Illuminate\Database\Eloquent\Relations\HasManyThrough
    public function hasManyThrough($related, $through, $firstKey = null, $secondKey = null, $localKey = null, $secondLocalKey = null){
        $through = new $through;
        $firstKey = $firstKey ?: $this->getForeignKey();
        $secondKey = $secondKey ?: $through->getForeignKey();
        $localKey = $localKey ?: $this->getKeyName();
        $secondLocalKey ?: $through->getKeyName();

        return new CustomHasManyThrough((new $related)->newQuery(), $this, $through, $firstKey, $secondKey, $localKey, $secondLocalKey);

Getting rancher/server back up after restarting from the web UI

If you restart the rancher/server container from within rancher it sets the state to shutting-down in the database, and then the container stops and the state cannot be updated anymore. If you try and start the server again it will see the state is shutting-down and shut itself down again.

To resolve, from the command line start a new rancher/server container with the same port and database config (but a different name; or let docker pick a random name), go to the web UI and in the Infrastructure/Containers list find your original rancher/server container and try to start it. It will fail (because the port is already allocated), but the state in the database will be updated. Then back on the command line, docker kill the new rancher/server and start up the old one (make sure the name is the same).

That should be it! The rancher/server should successfully be running again.

Connection aborted error using python elasticsearch with large files on AWS ES

AWS ES has an upload limit of 10MB. If you are using the bulk helpers or reindex and some documents are above this limit you will get an error ConnectionError: ('Connection aborted.', error(32, 'Broken pipe')).

To solve it, use the max_chunk_bytes argument, which can be used with reindex like so:

es, source_index, target_index, chunk_size=100,
bulk_kwargs={'max_chunk_bytes': 10048576},  # 10MB AWS ES upload limit

Ideally make the chunk size the average number of documents before the size is 10MB, and then in the case there are some larger documents that push the size over 10MB the elasticsearch library will handle it.

Error running vagrant provisioner when default vagrant share is disabled

If you have disabled the default vagrant shared directory

[sourcecode language=’ruby’]
config.vm.synced_folder ‘.’, ‘/vagrant’, disabled: true

You will get the following error using a provisioner:

==> default: Running provisioner: ansible_local…
default: Installing Ansible…
default: Running ansible-playbook…
cd /vagrant && PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ansible-playbook –limit="default" –inventory-file=/tmp/vagrant-ansible/inventory -v /ansible/init.yml
bash: line 3: cd: /vagrant: No such file or directory
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

To fix, set the provisioning_path configuration:
[sourcecode language=’ruby’]
config.vm.provision “ansible_local” do |ansible|
ansible.provisioning_path = “/ansible”

Importing/restoring elastic search snapshot to AWS Elastic Search Service

Took me a long time to find out how to do this.

A few people have re-posted a lot of this AWS article but missed out some crucial details:

The general idea is:

  1. Create an AWS bucket and put the snapshot files into it (don’t use a subdirectory, the .dat files should be in the bucket root).  No need to change permissions on the bucket or anything.
  2. Create an IAM role and policy as per the documentation in the AWS docs link above.  When creating the role using web management console you need to choose EC2 role type and manually modify the trust relationship after creating it.
  3. Run a python script (can find this in the docs link above) using the boto library to register the bucket as a snapshot repository in ES.  You need to sign the request regardless of the ES access policy you are using.  HOWEVER set `is_secure` to `True`.  Without this I was getting `<html></html>` returned instead of any error messages.
  4. Use curl to do the restore (no need to sign restore/backup requests if your access policy is open / IP-based).  Again check the doc for the exact curl command, but as above use https instead of http to get real error messages.