I'm looking for a something that allows me to automatically glean a final fully-qualified URL from a < FORM > on a web page. I don't care if it's an IE plug in, a Java applet, a third party program, whatever. As long as it does the job. Anyone know of one?

More detail:

When you have a web page with a <FORM> on it, that when you click on that final <INPUT TYPE=SUBMIT> button, it basically fashions a URL that contains all of the form's data fields concatenated together with & signs between them.

I often create actual bookmarks with the form data pre-filled-out. For instance, instead of having to fill out the username and password to a site, it's often possible to make a URL along the lines of:

http://www.somesite.com/cgi-bin/login.asp?username=myname&password=mypass

I use this method to log in to this BBS, so that I never have to type my user name and password, even if I've cleared all of my cookies, something I do often. (For those who are interested, the format for the current BBS software is http://empeg.comms.net/php/start_page.php?Loginname=yourname&Loginpass=yourpass&option=Login )

Sometimes, though, it's very complicated to figure out just what that URL needs to be in order to work. It involves digging through a bunch of HTML source code of the original login page and trying to glean which fields are necessary. Often, it requires many parameters on the command line which are hidden or generated by javascript code.

I'm looking for a tool which automates this task of digging through the HTML source code. I'd like to be able to go to a login page, type in my user name and password, but instead of hitting SUBMIT, I would instead run this tool, and it would parse the page's source code (or perhaps even proxy the SUBMIT request, that's probably even easier) and then present me with a nicely formatted URL that I can turn into a bookmark.

Does such a tool exist?

Huh, I've never run into anything close to this, so i'm not sure if it exists, but...

I'm pretty sure the only way this is going to work is if you happen to be trying a form with a GET method. CGI forms can be either GET or POST. Post ones don't show up in the URL(and should be used for passwords for just that reason)... If it's a GET method form, the URL should just show up in your browser...

Matthew

If it's a GET method form, the URL should just show up in your browser...

Often, the resulting page refreshes or redirects, so that even if the URL were available, I couldn't see it.

There's gotta be a way to do this with an IE plug in or something that captures the destination somehow... Maybe I'll mess with it in a VB applet and see if I can make it happen myself.

Hmmm... well, if you've got your own server, you could try just changing the "action" field in their form to point to your server, then look at your error log and see what URL was requested? I'm not sure how apache/your-web-server-of-choice actually handles forms that don't exist, however...

Matthew

Im not exactly sure how to do this, but a problem exists that if the secondary page expects data in the POST format, it wouldnt work to send it in the querystring. For example, some pages will request data like this Var = Request.Form("variable") and others like this Var=Request.Querystring("Variable"). If the secondary page expects posted data, then the resulting URL method wouldnt work anyway. Sorry that just creates more problems for you, just wanted to post my thoughts in case you didnt think of it.

Thanks for the tips. I'll mess with it.

Can't help but wonder if someone else hasn't already written such a utility, though.

The login page here at the BBS is a POST form, but the action of that form is forgiving enough (actually, being that I write web server security software, I would say STUPID enough) to also allow these manually created GET requests. This is a very poopy idea.

I somehow doubt such a tool exists, in that it would have to read your POST body data (dunno if there are *.Script client-side ways to get that) and plop it onto the end of your URL. How many sites do you visit that actually allow these credentials to be passed clear-text in GET requests? That's just awful.

Use the password, Luke. And use IE's built-in password storage feature which makes logging in when the cookie expires not-such-a-pain-in-the-keester.

Yea I would be suprised if it didnt exist as well. Seems like it could be a cool little utility. Its actually right up my alley (im an internet programmer), I might be able to pull something like this off. I know i could do it if you are willing to have the FULL html path...ie all fields would show up in the querystring instead of just those that are required. The problem lies in finding which ones are required by the remote server. Since most of this kind of stuff is written using server-side languages it is tough to actually tell which ones are needed and which ones arent. I dont know, let me think about it a bit, and maybe I can pull something off.

I know i could do it if you are willing to have the FULL html path...ie all fields would show up in the querystring instead of just those that are required.

Yeah, that's what I'm talking about. I wouldn't expect an applet to know what works and what doesn't, I can do trial-and-error once I've got the URL in front of me. The problem is getting the URL in the first place.

I've done this sort of thing dozens of times "by hand", it's just tedious to do, and I was looking for anything that would automate it for me.

So you want this to work on any site? I don't know how.

But if you just want to bookmark each site, it would be reletively easy to just save the source on the login page, then change the METHOD=POST in the form to GET, like someone said, then just run the page from your hard drive and bookmark the URL after you login. It wouldn't be as quick as running a program to do it for you, but it'd only take a couple minutes.

OK, Fabris. I made a quick javascript program to do what you're looking for. It's .hta file to overide the browser's security limitations that won't allow you to modify document.whatever in one window from another window, so it will probably ask you to save the file or run it from it's location.

First of all it has 1 limitation: The login form has to be the first form on the login page, which it is on most site's login pages.

Other than that, just follow the directions and it should work. Feel free to modify it and fix it up.

By the way, I guess I should mention that I only tested this IE 5.5.

Actually, let me clarify something. Contrary to the directions I gave in the program, you don't have to enter your name and password before you press the submit button in the program. You can press it before if you want. It doesn't matter, either way will work. You just have to wait until the page is loaded before you press the submit button.

Nothing big, I just like to be precise.

Yes, that seems to work, thanks.

Doesn't work for one particular site I'm messing with, but that's probably because it's one of those sites which doesn't let you mix up GET and POST queries.

I also didn't know about the trick with HTA files to make it get around the Javascript security limitations. I'll have to try that with some files of my own and see if I can work this functionality into something else I'm messing with right now.

Thanks for the info.

You can also embed javascript into URLs in favorites to get around that limitation. Create a new shortcut in the favorites menu and make the URL for it something like:

javascript:document.location.href = 'http:\/\/www.norman.cx/'; alert('boo');

Saving the favorite and then clicking on it runs the code within the context of the current page.

Thanks again for the code, I've improved on it a bit and now it will turn EVERY form on the targetpage into a GET by using the following code:

function changeMethod(){

  numForms = NewWin.document.forms.length

  for (var i = 0; i < numForms; i++)

  {

  NewWin.document.forms[i].method="GET";

  }

}

Okay, next two questions:

1) Is there any way to make IE quit complaining about the HTA file? Every time I open the HTA file, it says "Do you want to open this from your current location, or do you want to save it to disk?" and doesn't let me turn off the prompting. I assume this is a security feature.

2) Some of the destination pages... The initial URL that we feed it works, even as a GET. OK, so far so good. But then they do some kind of a refresh or redirect/reload of the page that appears. The perfectly good formatted URL that I wanted to copy suddenly vanishes from the page's address bar, before I have a chance to grab it. Is there any way I can prevent this refresh from happening on the targeted page? For instance, disallow javascript on the NewWin we just created?

Maybe it'll show up in IE's history ... uhhh ... panel thingy?

I can never get IE's history working right for me. When I click the little down-arrow, I expect to see a list of previous URLs, but instead I see the list of drives under "My Computer".

And besides, even if I could, how would I know which one was the right one?

Because you can sort the history to have it show you the pages in the order you visited them in. Maybe you're not thinking of the same history mechanism. I'm talking about the one that can be accessed via Ctrl-H or clicking the little sundial icon on the toolbar.

Ah, so that's what that button does.

You could have it document.write(NewWin.document.location ) in the program so it will display it right there. You could put it in a button and if you can time it just right, so right when the desired URL loads, hit the button, and it'll write the URL in the HTA window. I attached the program again with this button added.


And yeah, that prompting is a security feature, but it doesn't prompt me when I run it from my hard drive. I think it may prompt me only the first time I run it.
Here's some links about HTA's:
http://msdn.microsoft.com/workshop/author/hta/overview/htaoverview.asp
http://msdn.microsoft.com/workshop/author/hta/reference/hta_ref_entry.asp
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q241754

it doesn't prompt me when I run it from my hard drive. I think it may prompt me only the first time I run it.

It prompts me every single time I run it from my local hard drive. Rather irritating. Wonder how to work around that?

Take all the javascript in the hta, turn it into one long string of script, put a "javascript:" on the front and embed it in an IE favorite entry.

Unfortunately, it's not all javascript, it's partially HTML and forms.

From http://support.microsoft.com/default.aspx?scid=kb;EN-US;q241754 :

Cross-frame scripting was disabled to prevent a Trojan Horse-style security hole called frame spoofing, in which a Web site masquerades as a frame inside of a trusted Web site in order to steal user information. The following Knowledge Base article explains how to enable cross-frame scripting using document.domain for documents from different machines on the same network:

Q167796 PRB: Permission Denied When Scripting Across Frames

However, this is not effective for machines on the same network that resolve to different top-level domains; for example, Web sites on the same machine that use different virtual hosts. It also does not work for two distinct sites that have partnered with one another and wish to interact via frames.

In these situations, developers building solutions for Internet Explorer 5 can bypass cross-frame security for a frameset they have authored by turning it into an HTML application (HTA). This involves giving the page an extension of .hta and inserting the tag <HTA:APPLICATION> at the top of the page underneath the HTML tag. Each frame that attempts to script another frame (for example, the script source) must have the attribute pair "APPLICATION=yes" inside its FRAME tag.

When a user accesses the HTA, it asks whether he or she wants to "execute" the file. If the user says yes, the HTA opens in its own window. From that point on, documents can script freely across frames whose documents come from different domains. This is considered secure because it uses trust-based security: The user must verify that he or she trusts the host will refrain from intentionally malicious programming.

Another thing you can do that will also work with POST forms that require POST is..

- Grab the source of the page and put it in your favorite HTML editor (DreamWeaver, Notepad, whatever)
- Change the action into an absolute url
- Enter the data you want to submit into the fields
- Note the form tag's name. Give it one if it doesn't have one already (e.g. name="myForm")
- Put onLoad="myForm.submit()" in the body tag
- Save and put the HTML file somewhere.

Now if you load that file (with JS enabled), the form, with the content you've entered in it, is submitted exactly like you were at their site typing it in.

Yeah, I know you can do that, but I was looking for something where I could (a) generate JUST a URL, and (b) NOT have to muck about in the HTML source code.