The making of the Holy Grail or the Auction Search Engine

I have been partaking and watching some of the discussions on the making of the new MOAAS, (A Search Engine for Auctions that showed promise but suddenly faltered with no word of why.)

But some people seem a little lost in what we are asking of the search engine. I will try answer as best I can.

A normal search engine like Google and yahoo do not really go searching the web every time someone asks them to find something. They have already found it before you asked by sending out small programmes called spiders or bots that search the web scanning every site they find and mapping out the important details, and all that info has to be stored on a massive database or should I say lots of massive databases all over the world.
The reason they are so up to date is they have a lot of spiders running twenty four seven, as spiders are what they call low impact, which means they only use up small amounts bandwidth so they do not interfere with the running of the website.

They also have things like google base that you may know as google products search. Where people submit the info for their products, or the auctions they are selling on hand in the info for them. But this is only for items that have a fixed price and some auctions will only hand in items that fit specific criteria.

So when you ask for the latest mobile phone and google search for it, they are searching the database not the websites.

This is not really great for what we want but could be used for some parts of the search, such as fixed price, as long as we had enough spiders to keep it up to date. If the item is no longer available every time a customer searches for it then, they will stop using the search engine.

There are also other methods to search sites. You may have heard of meta search, that is closer to what we want but still has problems as it can produce massive amounts of data and if you look at sites like AuctionLotWatch that use this method, they will only scan the first page due to the amount of data it creates and the fact live searches can be slow if it tries to take on too much.

There are also the JavaScript searches that you can see with Alfie search or the Ebid searches that you can find on the PSU. These work by making use of the sites own search commands as if you were on them.

Problem with those is that not all sites can be searched, as some have coded search commands, it is limited by the number of sites that can be opened at once, it is controlled by the auction sites own options, there is no real sorted comparison and finally not everyone will allow JavaScript on their computer due to the fact it is heavily used in popups and can have security risks.

So what is the answer?

For me the perfect search engine (if we can’t have the auctions extending their own search engines) will scan the auctions like javascript but show the results in the same way google or a comparison script will show from the database.

But we can’t have that, as like I said live search is too slow at the moment but we could possibly request a compromise if we can get the auctions on board and try to get them to make a standard search criteria of search options. BIN, Auction, Ending Soonest, Newly Listed and all the other things we are used to, or more importantly our customers are used to. This would give the search engine a frame work to search under and the customers the option to thin down their search

Then after the user selects their preferred criteria we have a search engine that searches the first couple of pages the same way AuctionLot watch does. But at this point the server in normal circumstances will become dormant as the user reads the data it waits for the user to start pressing buttons or links.
So what if the Search Engine used this time to start searching on the next two or three of pages building a mini cache for the user sort of a smart search.
If we also had spiders doing the work, then some of the items could be drawn from the main database if there are a lot of fixed price in the search then all the better.

Now there have been a few other questions on why the search sites that are available now are not up to the job and do not seem to be covering the bigger auctions.

Again I can only offer what I see and make an educated guess but it could be a few things.
Database issues they just do not have the space.
Cost Issues, they have tried to charge for the service and failed.
Spider Issues, the spiders can be prevented from searching a site.
This can happen if the spider is not acting in the right manner, sticking its nose where it is not supposed to be, or is drawing on resources, as the person has set the spider up wrong, or as in ebays case with one company, the site thinks the search engine is detrimental to its business.
The only other issue that I can think of but there are probably a lot more is they are not searching the sites and are only using file upload like google product search, which is no use for auctions.

What you can do now if you are a seller?
If you sell fixed price items and you do not fit the criteria of your auction (ask them first,) then you could start making use of google base and sites like bidfind that will allow you to upload a csv file with all your fixed price items. If you do not have fixed price items, then would it not be possible to add a few just to make use of this service. And draw customers into your other products.

I think I have to stop working on this now for a little while, as it has started to cost me money, while I am doing the auction search stuff I am getting behind in the listing of items and stuff. I still have not had time to list on the new auctions, which I have to get round to as you see in the countdown on the top time is running out.