How to Fool Some of the People Most of the Time.

May 1st, 2009 by greentheo

We all know that monkeys (or darts) can pick stocks just as well as most fund managers.  Why? The amrkets are very close to being efficient.  Thus any fund manager, in theory, has no advantage over any other unless he knows something the markets don't (and most of the time they do).

But we do know that certain managers seem to consistently pick good stocks... so do they know something everyone else doesn't?  Probably not.  In a pool of 1,000 fund managers we would expect that a handful will have a great record, and a handful will have a terrible record... but most will be average... even if they all pick their stock portfolio's at random.

The human mind, being a great pattern recognizer, has a terribly difficult time accepting the fact that good performance over an extended time period (for a complex prediction problem like the stock markets) may just be a product of randomness.

And thus here is a wonderful method for fooling some people most of the time - the example is for convincing people to purchase your pick for this year's super bowl winner:

  1.  Start off with a list of 132,000 people for each football team.
  2.  Week 1 of NFL football, send out predictions for each team to the corresponding list.  For each team make 50% of the predictions on the list wins, and the other 50% losses.
  3. Week 2 of NFL, for those on the list who had a correct prediction, repeat step 2.
  4. Week 3 - 16, repeat step 3 taking the previous week's winners and assigning a random prediction.
  5. By week 16 (the super bowl), there will be 16  teams playing and for each team you'll be left with about 1 person for whom your prediction has been correct every single week.
  6. Offer each of the 16 people the chance to buy your predictions for the playoff at 100 dollars the first round, 500 dollars the second round, $1,000 dollars for the Championship, and $5,000 for the super bowl.
  7. Assign a win prediction to each person left, as their team cruises towards the Super Bowl the will more and more excitedly hand you their money (and gamble off their life savings).
  8. After the Super Bowl offer the winner the chance to purchase next seasons predictions for the low price of $1k, but only if he signs up each of his friends for $1k each.
  9. Take your $40k (assuming you get your Super Bowl winner and 10 of his friends to purchase for next year), and head to Mexico... try not to get Bernie Madoffed.

In the world of stocks the above example is equivalent to having a 20 year track record of excellent returns!

Now you tell me... how much of your hard earned money do you want to invest in my mutual fund? Really... with a track record like ours... how much could you possibly lose?

Tags: , ,
Posted in 1 | No Comments »

Inefficient Market Hypothesis

December 27th, 2008 by greentheo

The financial markets are touted as being efficient by most leading economists.  A large faction of economists tend to disagree and claim that the markets are only mostly efficient because humans are not fully rational agents.

An efficient market is one where all available information is accounted for in the price of an assett or good.  If there is a deal to be found, then it has already been found by someone else.   For instance, car dealership XYZ misprints a coupon and offers a $50k BMW for $25k to the first buyer.  An efficient market hypothesis formulates that someone else will arrive as quickly as possible to snatch up the good deal.

An inefficient market is one where prices do not take into account all available information, or one in which the participating agents don't make the most logical decisions.  As in the above example the dealership offers a brand new BMW for 50% off... a great deal!  But suppose they make the offer on Saturday in a 100% Orthodox community.  Orthodox Jews can not handle money on the Sabbath and thus the BMW goes unpurchased until Sunday leaving the random out of town straggler to stumble upon the deal of a lifetime.

Aside from quirks like the above, the markets are usually efficient.  If there is a deal to be had, it will be had. But I'd like to formulate my own hypothesis on inefficiency which stems from my understanding of the human psyche.

Markets are inefficient because human decisions are influenced by the following: overestimated future extraction of the good of an asset, greater fear of loss than hope at an equal sized gain, pride of ownership.

1.)  Humans are extremely biased creatures.  We have to be.  In order to survive it's necessary to make decisions without all of the data.  This means bias.  When it comes to a purchase or investment humans continue the pattern.  Many of our purchase or investment decisions are made by comparing a list of highly complex options.  Once the decision has been made (just go for it) the mind then plays the trick of soothing the resulting anxiety (did I make the right decision?) by hyping up the enjoyment that the purchase or investment will bring in the future.  As the future has not arrived yet it's perfectly understandable to assume that product X or investment Y will bring us more fun or wealth than we ever imagined.  This is not rational and leads to inefficiencies in the market.

2.) Humans fear loss far more than they hope for an equal sized gain.  If you were to take a survey of 1,000 people and ask them to value what their right arm was worth, take the average amount and triple it you'd still find that almost all of those surveyed would still rather keep their right arm than exchange it for the money.  In another example we can take marriage.  A young man dating a beautiful young woman often balks at the idea of marriage until he loses the potential bride, simply because the fear of the loss of other potentially better brides is greater than the hope of living a long and happy life with a beautiful and smart woman[1].  We hate risk, irrationally so, and it causes inefficiencies in the markets.

3.)  Humans love to be owners.  And once we own something, our mere ownership causes the item to become more valuable.  Our pride in ownership is purely an emotional state which is the result of our inborn need to nurture.  We are built to love, nurture and cultivate.  It's how God (or Evolution if you insist) designed us.  Give us a baby, we feed it, a dog and we pet it, a plant and we water it, a car and we change the oil, a house and we fix it, some stock in a company and we watch it "grow".  It is this irrational attachment to our friends, children and even things that has allowed us to continue from generation to generation, but it also causes inefficiencies in the market place.

What does this all add up to?  It adds up to two consequences:

Inefficiencies cause bubbles and bursts...mass movements towards and away from investments and goods fueled by irrationality. The bad thing about inefficient markets is that you may get caught in the burst unprepared.  But the great thing is that inefficient markets provide opportunities for gain.

The main lesson is don't be afraid about an economic crisis... it's irrational... and besides it's likely that another bubble is forming somewhere.  If you're quick you can probably catch it on the upswing!

  1. I also wonder, from an economic perspective a good woman is worth far more than her weight in gold, quite literally, why then would a man prefer to live the single uncommitted life? [↩]

Tags: , , , ,
Posted in Math, Miscellaneous, Stock Market | No Comments »

Hedging your investments... explained

November 12th, 2008 by greentheo

When most think of hedge funds they think of big fat cats sitting around in some office with billions at their disposal rolling the dice on risky futures and derivatives.   While this might be true in some cases it doesn't explain what a hedge, or a hedgefund does.  And let's start by eliminating the obvious... a hedgefund does not invest in thick bushes that line driveways :-)

Any investment can be considered a hedge if it helps to reduce the risk of overall loss due to a seperate (usually larger) investment.

The best way to think about a hedge investment is with an example that we should all know very well, cars and car insurance.

When one purchases a vehicle there are two financial sums at risk... the vehicle itself and the damage caused to others or yourself in an accident.  It may happen that while I'm driving my $50k hummer down the highway I, oblivious to the rest of the world, crash into another Hummer (who is equally oblivious to the real world).  Since the accident is my fault I'm now on the hook for my hummer, the other hummer and all medical bills... let's say $200k all together.  Luckily I hedged against catastrophic loss by taking out a comprehensive auto-insurance policy for $100 a month... they pay for everything.  I'm free to return to my irresponsible hummer driving ways!

Notice that for $100 a month, my hedge has the ability to pay out $200k when a predefined event occurs, this is called leverage: the ability for a small sum of money to control a large sum.

Hedging is not limited to insurance though.  In the stock market one can easily hedge against significant loss via Put and Call options.  If I sell a Put option to some other investor, we've agreed that I can "put" my stock on him (sell it to him) for an agreed upon price.  In exchange for the risk that the Put option buyer takes on, I agree to pay him a small fee...say 5% of my stock's value.  When my stock shoots up 10% the put option gives me no advantage and I keep my stock.   But perhaps it loses 20%... I then shovel off the stock to the purchaser of the put contract for its agreed upon value.  Now instead of losing 20%, I've only lost 5%.

Notice:  5% of the stock's original value creates a contract to control 100% of it's value.  Again, we see leverage here.

A Call option works the other way, it is an agreement between me and a holder of some stock such that I can "call" him up and purchase the stock for an agreed upon price.  In this case I pay the owner 5% to buy the stock at a certain price.  When the price of the stock shoots up and above our agreed upon price by 20% I excercise my option to buy and purchase the stock for a 20% discount.  If the stock fails to increase in value the owner of the stock makes a 5% profit and I lose the nominal value of the call option.

Again, for 5% I can gain control of 100%.

The essential feature here is that I can take on another's risk or dish out some of my risk for profit.  No problems here with massive default.

However, what if I don't have stock to sell options on or the money to purchase the asset indicated in the contract?  I can go naked! wooohooo!!

No not that kind of naked.  Going naked means that you sell someone the option to "put" stock on you without having the money to pay for that stock (which would cause a default on your side).  Likewise you can also sell the right for someone to "call" you on your stock.  In both cases you collect the 5% fee, but you'd better hope that the conditions don't force you to to have to purchase the requisite amount of stock in order to fulfill the contract[1].

Selling naked options is not hedging, it's highly leveraged speculation and is ill-advised.

Hedging is not limited to what has been shown above.  Almost any investment can be a hedge as long as it protects against loss from another investment ... the possibilities are almost limitless, for example: biodiesel hedges against rising oil, hay bails against rising drywall costs, Japanese steel against American Steel, Apple vs. Microsoft etc.

Perhaps one of the most common hedges in the average person's life (besides car insurance) is the common resume and occasional job application.  That's right, hedge against income and job loss by keeping one foot in the door of another company... it only takes a little bit of your time.

  1. this is called securities fraud and => jail [↩]

Tags: , , , , , , , , , , ,
Posted in Finance, Math, Stock Market | 2 Comments »

When computers control all essential societal functionsw

November 3rd, 2008 by greentheo

It is apparent to a researcher in Machine learning and Artificial intelligence that soon enough machines will be controlling many of the essential resources of our society.  The webs that will be woven by the interdependent parts of our cities and countries  will be held together by increasingly powerful and complex networks of computers.

Since computers can store far more information in their immediate memory and can play out 1,000's of scenarios per second they are the clear choice for such jobs as: air traffic control, airplane scheduler, train engineer, traffic and road designer, stop light designer, telephone switch operator, and any number of jobs that require such comlex decision making.

I don't think machines will ever "take over" like they do in the matrix, but the question will soon arise, what happens if we unplug one machine?  What about 20?  Furthermore, what if the machines make apparently sub-optimal decisions... how will we be able to tell if they really were sub-optimal?

I think the most likely worst-case-scenario will begin to be not that the machines take over the world and force us into slavery, but rather that our dependence on them becomes so great that we are afraid to do anything without their consent... even if it means doing something that is right, but crosses logical boundaries.

Posted in Machine Learning, Miscellaneous | No Comments »

Problem with 'Table Full' for MyISAM DB?

November 3rd, 2008 by greentheo

If you need to expand your MyISAM table size for your MySQL DB becuase you simply have too much data then this is a nice quick tutorial on accomplishing your goal.

http://jeremy.zawodny.com/blog/archives/000796.html

Basically do the following

mysql> alter table your_table max_rows = 200000000000 avg_row_length = 50;


And that will increase it to its maximum size.



				


				
Tags: , , 
 Posted in Code, Miscellaneous |   No Comments »

			


		
			

				
mxmlc segfault Ubuntu 8.04

				October 24th, 2008  by greentheo

				

					
I got a segfualt error when running mxmlc on a new flex swf that I was trying to build.


Turns out the problem is that if you have recently updated your Ubuntu installation, it might have reverted to a previous java runtime environment.


To check your environment:


java -version


It should output:


greentheo@kingsnood3:~/Flex_Builder$ java -version

java version "1.6.0_07"

Java(TM) SE Runtime Environment (build 1.6.0_07-b06)

Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)


If you get a different version you'll want to switch it to the latest SUN sdk.


try: sudo apt-get sun-java6-jre


Once, it's installed, change the default java jre to the sun jre with:


sudo update-alternatives --config java


There are 3 alternatives which provide `java'.


Selection    Alternative

-----------------------------------------------

1    /usr/lib/jvm/java-6-sun/jre/bin/java

*+      2    /usr/lib/jvm/java-gcj/jre/bin/java

3    /usr/bin/gij-4.2

And choose the option with the Sun Java jre.


Now get back to work on your Flex application!

				


				
Tags: , , , , , 
 Posted in 1 |   No Comments »

			


		
			

				
Adaptive UI

				October 9th, 2008  by greentheo

				

					
Designing a website, or a User Interface for a program is hard work.  One has to study people and their often hard to predict and difficult to quantity decision making processes.


For instance, google.com shows a clear knowledge that user's get distracted by anything other than the task at hand if presented with multiple options.  Thus their site has one main focus... a search bar.  They, are are after all, a search company.


google.jpg yahoo-screenshot-2.jpg


But what if User Interface designers, and website designers started to pay attention to the actual math behind user interactions?  Like living organisms, what if the website or user interface adapted to the user?  For instance, if the website knew that people browsing on cloudy days liked a more cheerful design why wouldn't it brighten up the background?


Or perhaps in Yahoo's case why not reorder the sidebar links in order of most clicked on?  What about Yahoo's search bar?  Might that feature become larger and more prominent?


Why aren't websites and Program designers doing this already?  They may be, but the sheer volume and dynamism of large user bases makes it a very difficult and ever changing job.


I suggest that website and User Interface designers move to a design framework which includes the ability for the site or program to collect user statistics, and then reorganize itself according to some preset rules.  Flex, Flash, php, python and a host of other programming languages could all support these features really easily.  And if you can't think of a better optimization algorithm, simply allow the user interactions with the site to be a sort of genetic algorithm.

				


				
Tags: , , , , , , , , 
 Posted in Machine Learning, Miscellaneous, Statistics |   No Comments »

			


		
			

				
Local Search Optimization

				October 2nd, 2008  by greentheo

				

					
What do you do to find the optimal solution to a problem that has an overwhelmingly large amount of equally appealing solutions?  Well, it turns out that mathematicians and computer scientists haven't even really been able to answer the problem either.


In the book Combinatorial Optimization, Christos H. Papadimitriou admits that for most complex problems there is no easy way to find an optimal solution given a finite amount of computation time.  He recommends Local Search as a good starting point.


Local Search is actually a very simple algorithm which humans have been using for years. It goes as follows:


    

  1. Start off with your best guess, if you have no best guess then start off with a set of random solutions
    2. 

  2. Modify your initial solution set slightly.  I.e. move about a small neighborhood of your initial solution set.
    3. 

  3. If solution get's better, keep on pursuing the direction you moved in, else try another neighboring solution.
    4. 

  4. Keep proceding until a sufficiently optimal solution is found
    5. 



Imagine the problem in this way, your friend told you he was going hiking next week, but didn't tell you a day, time or location.  1 full week from now you've not heard from him, and are getting worried.  You call in the search and rescue team but where to start?  Colorado has thousands of trails.


Well, the best place to start is probably his house, then the 5 most frequently hiked or mentioned trails by your friend.  After that, go with the most popular, then the most available to hike.  If you haven't found him at the initial places, then look at the trails closest to the ones just searched.


Good ol' combinatorial optimization!

				


				
Tags: , , 
 Posted in Math |   No Comments »

			


		
			

				
Quantastic!

				September 26th, 2008  by greentheo

				

					
I am very proud to announce that my years (3) of research into the stock market have paid off.


I will now be working with Northfield Trading LP here in Denver to develop automated trading algorithms for futures markets.


Should be alot of fun, growth and learning.

				


				
 Posted in Finance |   No Comments »

			


		
			

				
A good analogy for Operations Research

				September 25th, 2008  by greentheo

				

					
The other day I was trying to explain the concept of constrained optimization to an artist.  As I was fumbling around a bit, my friend (who was also a Math nerd) said:


Constrained optimization is like fishing... you have a limited amount of bait, you don't know what the fish want, how many fish are out there, or what type/proportion they are but the goal is to catch as many as possible in a few hours.


 While it's not a perfect analogy... it's pretty good.  What's your favorite analogy for constrained optimization?

				


				
Tags: , , , , , 
 Posted in Math, Miscellaneous |   No Comments »

			


		
		

			
« Older Entries