Create a nonstandard region of hint

First of all, you should create a new class with ActiveHint procedure. When you create form, you should set HintWindowClass to your new class (TMyHint). Try this:

  TMyHint = class(THintWindow)
  private
    FRegion: THandle;
  public
    procedure ActivateHint(Rect: TRect; const AHint: string); override;
  end;

...

procedure TMyHint.ActivateHint(Rect: TRect; const AHint: string);
begin
  SetWindowRgn(Handle, 0, True);
  DeleteObject(FRegion);
  BoundsRect:=Rect;
  FRegion:=CreateEllipticRgn(0,0,Width,Height);
  SetWindowRgn(Handle, FRegion, True);
  inherited ActivateHint(Rect, AHint);
end;

procedure TForm1.FormCreate(Sender: TObject);
begin
  Application.ShowHint:=False;
  HintWindowClass:=TMyHint;
  Application.ShowHint:=True;
end;