The strange behaviour of wget in the night

I’ve knocked up a quick Delphi executable that does what I need to keep my email signatures bright and exciting.

The strange behaviour of wget in the night

A couple of years ago I wrote a piece about dynamic email signatures. Recently I migrated my website and my email signature code to a new server. During the migration I made the appropriate modifications to the email signature server file, tested it in my browser and gave it the big tick.

But today, when the migrated site went live, I discovered that it didn’t work. It seemed wget was unable to retrieve the output of the .php file even though it appears perfectly in my browser. After some head-scratching and investigation I found the reason in the server’s log file. Unlike my previous installation, this server was returning a 404 message when the .php file was accessed. It was also returning the file output, but wget wasn’t having a bar of that: it responded to the 404 message by discarding the output.

Others have struck the same issue: there are a number of posts on the web with users wanting to download the contents of 404 files and striking the same behaviour.

For the moment I’ve knocked up a quick Delphi executable that does what I need to keep my email signatures bright and exciting. It’s a trivial piece of code using one of Francois Piette’s excellent ICS components and a couple of utilities from my own toolbox. It runs silently in the background, and in that respect at least works better than the wget batch file I was using previously. Here’s the code.

program mail_sig;

uses
  SysUtils,
  Classes,
  Dialogs,
  Controls,
  OverbyteIcsHttpProt in '..\..\..\Shared\Components\Piette ICS\Delphi\Vc32\OverbyteIcsHttpProt.pas',
  WinUtils in '..\..\..\Shared\WinUtils.pas';

var
  FHTTP: THTTPCli;
  sFile: String;

begin
  if not FileExists(ChangeFileExt(ParamStr(0), '.txt')) then begin
    if MessageDlg('Please refer to the readme.txt file.', mtInformation, [mbYes, mbNo], 0) = mrYes then
        OpenWebPage('http://www.else.co.nz/');
    Exit;
  end;
  try
    Wait(1000);
    FHTTP := THTTPCli.Create(nil);
    try
      sFile := ChangeFileExt(ParamStr(0), '.htm');
      with FHTTP do begin
        RcvdStream := TFileStream.Create(sFile, fmCreate);
        URL :=  LoadStringFromFile(ChangeFileExt(ParamStr(0), '.txt'));
        try
          Get;
        finally
          RcvdStream.Free;
          RcvdStream := nil;
        end;
      end;
    finally
      FHTTP.Free;
    end;
  except
    on E: Exception do begin
      //
    end;
  end;
end.

Everything happens in the same directory, which is all I needed for this exercise. The executable expects to find a .txt file with the same base name in its directory, and it expects that .txt file to contain the URL it’s going to request. It outputs a .htm file in the same directory containing the downloaded web output. Of course, the executable and text file names can be changed to deliver whatever output filename you need (as long as it ends in .htm).

Some notes about the code.

  • OpenWebPage and LoadStringFromFile are simple utility functions in my WinUtils unit.
  • We wait for a second at the outset to ensure the email client’s loaded the existing output file before we get a handle to it.

On reflection I’m not sure I agree with wget’s logic. Particularly if I’m downloading multiple files I want to know about the failures: I don’t see that saving the output is a bad thing where a 404 error is returned.

Feel free to download the binary. It’s a bit bloated but it works well.

What do you think?

Comments are aggressively moderated. Your best chance is reasoned disagreement.

*