What type of file is it? Most files have a file name with a three or four letter extension which tells the machine what program it belongs to. For example, a .XLSX file is a Microsoft Excel spreadsheet file. This information is stored on your Mac in a table of File Associations. In most cases, the File Associations table is updated when you install a new program, and your Mac knows what files each program can handle. But sometimes, a new data file can fool the Mac.
Note: If the file is an .EXE file then it is a Windows program file and will not work on your Mac. (In rare cases an .EXE file is a self extracting archive file that can be opened with Stuffit or another archive program.)
If the file was made by a program that you don’t own on your Mac, then OSX won’t know what to open. Your Mac doesn’t have an association for the extension of the file (or the file has no extension) so it gives up and either says it cannot locate a program, or it tries to open it with TextEdit thinking it is a text file.
If it is a file type that you know you can open with one of your existing programs, then you can Right click (or control-click http://www.macinstruct.com/node/66) on the file icon, and choose Open With… and choose the program that you want to open this file with. If your desired program doesn’t show up on the list, click Browse… to find it.
If you want all files of this type to open this program in the future, after you have clicked Browse.. and found the program, click the box that says “Always open files of this type” and it will make that association permanent.
Alternatively, you can set a file association in the Finder by Right clicking on the file (or using File: Get Info or Command-I) and choosing a program in Open With… for that file. If you want all files of this type to open with that program, press Option after you open the contextual menu, but before you click on the Open With… this changes it to Always Open With…
If you don’t know what program made the file, or if it came from a Windows or Linux system, you may need to do some extra research to find out what software is capable of opening the file. Preview will open most graphic files, Photoshop can open files from more types of graphic programs, Word (or OpenOffice) can open many different word processing file types, Excel (or OpenOffice) can open spreadsheets plus many text-based data table and database types such a .csv, .tab, .sylk, .wks and others.
There was a program called MacLink Plus which did file conversions, but sadly it has been discontinued by the publisher.