At what portfolio size does it make sense to switch from holding index funds / ETFs to directly holding the indexed instruments?

If your time is worth less per hour than the fee you pay the fund manager for management, then the dollar amount invested has no bearing on the question; you should own/manage the indexed instruments directly.

If you are asking, it is never worth your time to do it yourself.  The tracking error and transactions costs are not worth it if you are running less than at least $5M for tax free funds.  If you have a taxable account, it may make sense with as little as $500K or $1M.  If you are starting with cash, it is probably still not worth doing yourself, assuming you are tracking a broad benchmark. However, in the unusual case that you have one or a few low basis stocks (say that your firm was bought by a Fortune 500 firm) then it can make a lot of sense to utilize intelligent tax-loss harvesting algorithms to diversify your portfolio, while avoiding a large tax hit.  This can add a lot of value, and it is the prime area where I would consider individual equities better than an index fund in a taxable setting.  Even in this case, their are several firms that offer these services including Aperio, State Street and Parametric. In short, unless you have at least several million $ and still think the best thing you could do with your time is to design a complex optimization scheme with integer constraints and wade around sorting out market data, just buy an index fund already.  If you are that eccentric and these kinds of problems sound like fun, please contact me.

Do you want an index fund or the market portfolio? If you want a market fund and you have a factor model or the CAPM is wrong then your market portfolio will be different than the index fund. In general, it does not matter how much money you have.  You can achieve extremely low transaction costs from places like IB.  Tracking error does not affect your rate of return much. In fact, your rate of return may be higher if you are investing in the market portfolio (as determined by mean-variance analysis and your factor model).  You will notice that in theory (CAPM) the market-fund and index fund are the same portfolio; however in practice they diverge significantly. I would definitely just calculate the market fund and track that.  There are many advantages to doing it yourself, especially if you want to employ tax-loss harvesting strategies.